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ABSTRACT 


Machine  grading  techniques  are  becoming  of  greater  importance 
because  of  the  increased  number  of  students  in  programming  classes. 
Characteristics  and  limitations  of  automatic  machine  grading  systems 
are  proposed.  A  grader  for  introductory  assembly  language  programming 
courses  was  developed.  The  properties  of  this  grader  are  discussed 
and  an  example  of  a  graded  program  is  given. 
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I.   INTRODUCTION 

A.  BACKGROUND 

The  capability  of  an  instructor  to  teach  programming  courses  and  to 
grade  student  programs  in  classes  with   large  numbers  of  students  has 
been  significantly  enhanced  by  automatic  machine  grading  systems.     The 
concept  of  computer-controlled  grading  of  student  programs  was  first 
documented  by  Hollingsworth  111    in  1960.     Machine  grading  systems  are 
known  to  exist  at  Rensselaer  Polytechnic  Institute  111   ,  Stanford 
University  I  2,3     ,  and  at  universities  in  England  and  Australia. 
These  grading  systems  provide  a  method  of  objectively  grading  student 
programs  automatically  for  the  instructor.     By  utilizing  the  computer 
as  an  extension  of  his  teaching  methods,  an  equal   amount  of  attention 
in  evaluating  class  programming  work  is   possible. 

B.  OBJECTIVE 

The  objective  of  this  thesis  is  to  investigate  the  concept  of 
machine  grading  systems  and  to  provide  a  practical  implementation  of 
a  grader  for  beginning  assembly  language  students.  The  fulfillment 
of  this  objective  is  accomplished  by  an  analysis  of  grading  tech- 
niques and  parameters  available  for  evaluating  a  program.  The  second 
element  of  this  objective  is  accomplished  by  establishing  an  operating 
system  specifically  designed  for  assembly  language  instruction  at 
the  Naval  Postgraduate  School  and  constructing  a  machine  grading  pro- 
gram which  will  operate  within  this  system  and  produce  graded  student 
programs.  Through  the  courtesy  of  Professor  Andries  van  Dam  of  Brown 
University,  the  Brown  University  Student  Operating  System  was  made 
available  to  the  Naval  Postgraduate  School  for  use  in  the  instruction 
of  assembly  language. 


C.   METHODOLOGY  AND  ORGANIZATION  OF  THESIS 

The  methodology  of  this  investigation  was  to  first  review  previous 
work  in  the  field  of  machine  grading.  The  metrics  of  a  program  which 
can  be  utilized  in  its  evaluation  are  discussed  in  Chapter  II.  A  sum- 
mary of  the  essential  properties  of  an  automatic  grading  system  is 
presented  at  the  end  of  Chapter  II.  The  description  of  a  grading 
system  and  its  implementation  is  given  in  Chapter  III.   In  addition, 
grader  use  with  student  programs  is  discussed.  The  conclusions 
reached  after  the  investigation,  and  the  recommendations  for  further 
study  are  contained  in  Chapter  IV. 

The  first  step  in  constructing  a  grader  system  was  to  revise  the 
structure  of  the  Brown  University  Student  Operating  System  and  im- 
plement it  at  the  Naval  Postgraduate  School.  The  Student  Operating 
System  was  then  examined  for  inherent  characteristics  which  provide 
the  instructor  with  teaching  and  grading  capability.  An  appraisal 
of  this  system  with  respect  to  grading  capabilities  is  presented  in 
Appendix  A.  The  other  appendices  contain  various  computer  programs, 
flowcharts,  and  outputs  relating  to  this  study. 


II.   CHARACTERISTICS  OF  A  GRADER 

A.   REVIEW  OF  AUTOMATIC  GRADING  SYSTEMS 

A  search  of  the  literature  for  reports  of  automatic  grading  tech- 
niques produced  several  documents  specifically  addressing  the  subject. 
Perhaps  the  first  published  report  of  grader  programs  is  that  of 
Hollingsworth  [l]  .  This  grader  was  used  at  Rensselaer  Polytechnic 
Institute  in  1959  and  was  implemented  on  an  IBM  650  computer.  The 
author  states  that  the  university  could  not  have  accommodated  pro- 
gramming classes  of  80  to  120  students  without  the  use  of  the  grader. 
This  early  version  of  an  automatic  grader  provided  batch  processing 
and  grading  of  student  jobs  for  a  variety  of  exercises. 

Students  submitted  new  programs  and  corrections  to  old  programs 
for  keypunching  at  the  end  of  a  class  period.  After  keypunching,  the 
corrections  were  filed  with  the  old  programs  and  all  the  programs, 
old  and  new,  were  arranged  by  exercises.  The  grader  program  deck 
was  inserted  with  the  student  jobs.  Considerable  effort  in  hand 
filing  punched  cards  was  necessary  in  this  system. 

This  grader  punched  identification  cards  for  student  programs, 
checked  results  and  punched  either  WRONG  ANSWER  for  an  incorrect 
solution  or  PROBLEM  COMPLETE  for  all  correct  answers. 

A  great  deal  of  operator  control  was  required  to  process  student 
programs.  Manual  re-entry  for  overflow,  invalid  addresses,  and 
other  problems  was  required.  A  limitation  of  this  grader  was  the 
fact  that  it  processed  only  machine  language  programs.  Student 
programs  could  modify  the  grader  itself,  causing  additional  problems. 


The  grading  system  required  the  specification  of  four  addresses  associ- 
ated with  the  exercise,  viz.,  the  location  of  the  entrance  to  the  stu- 
dent program,  the  location  to  which  the  student  program  must  exit,  the 
address  of  variables  and  the  address  of  results. 

An  advantage  of  this  system  was  that  programming  classes  were  not 
required  to  work  on  a  single  project  at  the  same  time.  Since  various 
exercises  could  be  graded  during  one  batch  run,  students  could  work  on 
several  projects  simultaneously  if  they  so  desired.  Despite  some  dif- 
ficulties with  this  system,  Hollingsworth  I  1 J  indicated  that  the  grader 
had  more  than  justified  the  effort  in  its  implementation. 

Automatic  machine  grading  programs  in  use  at  Stanford  University 
are  reported  by  Forsythe  and  Wirthl  2,3  I  .  Two  ALGOL  grading  programs 
have  been  used  in  connection  with  introductory  ALGOL  programming  and 
numerical  analysis  courses  at  Stanford  University  since  1961.  One 
grader  simply  furnishes  data  and  checks  answers.  The  other  program 
provides  a  searching  test  of  the  reliability  and  efficiency  of  an 
integration  procedure.  The  major  difficulties  and  limitations  of 
the  Rensselaer  grader  were  overcome  in  the  grading  systems  in  use  at 
Stanford.  The  concept  of  basing  a  grade  only  on  the  binary  answer 
"CORRECT"  or  "WRONG"  was  used  for  beginning  students.  More  advanced 
students  were  graded  by  the  second  grader  on  their  ability  to  solve 
the  integration  problem  efficiently  and  with  some  degree  of  reli- 
ability. The  authors  claim  that  this  grader  marks  a  further  step 
in  the  automation  of  grading  because  the  quality  of  the  program  is 
evaluated. 

R.  E.  Berry  4  examined  the  use  of  automatic  grading  programs 
for  checking  the  practical  work  of  students  in  a  numerical  analysis 
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course.  This  report  provides  a  comparison  of  the  Forsythe  and  Wirth  pro- 
gram I  2  and  a  grader  program  by  Naur  I  5  J  .  Both  of  these  graders  con- 
sidered the  same  problem  (finding  the  root  of  a  given  function  to  a  given 
accuracy  in  a  prescribed  interval). 

Naur's  method  required  that  the  student  program  be  included  in  the 
grader  program  as  a  labelled  block  of  coding,  accessed  by  means  of  a 
switch.  The  grader  assigned  values  to  variables  which  were  global  to 
the  block.  In  this  manner  different  problems  could  be  presented  to  the 
student  algorithm.  When  a  solution  was  achieved,  a  branch  to  a  given 
label  within  the  grader  program  was  made  and  an  examination  of  the  re- 
sult was  conducted.  Further  problems  were  then  presented  until  the 
problem  list  was  exhausted  and  the  grader  continued  with  the  next 
student  program.   In  contrast,  the  ALGOL  grader  implemented  at 
Stanford   2  was  called  by  the  student  program.  The  students  were 
required  to  write  a  procedure  with  parameters  in  a  specific  order. 
Each  procedure  was  tested  by  supplying  it  with  different  sets  of 
parameters  and  checking  the  solution  obtained. 

One  of  the  difficulties  of  grading  was  well  illustrated  in  the 
work  by  Berry  I  4J.  The  problem  of  evaluating  the  quality  of  an 
algorithm  for  a  problem  which  has  various  methods  of  solution  is 
exceedingly  difficult.  The  author  states  that  in  trying  to  present 
a  problem  to  which  different  methods  of  solution  are  available,  the 
possibility  of  successfully  grading  the  work  presented  is  remote. 

Berry  points  out  that  the  demand  for  automatic  grading  programs 
in  Great  Britain  is  not  expected  to  be  substantial,  since  computer 
classes  are  smaller.  One  commendable  aspect  of  graders  was  verified 
by  both  Naur  [_5J  and  Berryl  4j.  They  arouse  interest  and  provide 
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an  incentive  to  work.  Students  submitting  programs  to  this  type  of 
grader  found  the  challenge  of  writing  the  best  procedure  an  incentive 
in  their  work. 

The  first  known  grading  procedure  for  PL/1  exercises  was  implemented 
on  an  IBM  System/360  model  50  computer  at  the  Australian  National  Uni- 
versity in  1966  I  6  I.  The  decision  to  use  automatic  grading  by  com- 
puter of  all  student  attempts  at  class  problems  was  based  on  the 
fol lowing: 

1.  The  large  volume  of  work  required  for  teachers  to  carry  out 
grading  and  associated  record  keeping. 

2.  The  apparent  difficulty  of  teaching  input/output  sufficiently 
early  in  the  course  for  students  to  begin  writing  programs, 

3.  The  ease  with  which  input  test  values  could  be  supplied  to  the 
students  and  results  printed  under  control  of  a  grading  procedure. 

The  idea  of  using  a  grading  system  to  bypass  difficult  program- 
ing aspects  during  beginning  phases  of  instruction  was  not  found  in 
earlier  documentation  of  graders.  Temperley  and  Smith  [6  J  also  point 
out  that  the  use  of  PL/1  made  possible  the  evaluation  of  results  in  a 
variety  of  formats.  Test  data  and  results  could  be  in  the  form  of 
arrays,  structures,  bit-strings  and  character-strings  as  well  as 
scalar  numeric  values. 

The  timeliness  of  this  investigation  is  borne  out  by  a  recent  paper 
on  an  automatic  grading  scheme  for  simple  programming  exercises  by 
Hext  and  Winings  7J.  This  report  documents  the  development  of  the 
Basser  Automatic  Grading  Scheme  (BAGS)  at  the  University  of  Sydney, 
Sydney,  Australia.  A  major  difference  in  the  BAGS  system  is  that  it 
can  handle  exercises  in  several  different  languages.  No  special 
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action  by  computer  operators  is   required  with  this  scheme.     Another  sig- 
nificant departure  from  previously  documented  graders  is   the  structure 
of  the  system  itself.     Whereas  most  other  schemes  embed  their  exercises 
in  some  larger  program,  BAGS  is  part  of  the  standard  operating  system 
and  its  exercises  are  run  under  batch  processing  methods. 
The  basic  requirements  of  the  program  were  as  follows: 

"    (1)     It  should  handle  exercises  in  ALGOL,  in  MINIGOL   (a  subset  of 
ALGOL)   and  in  KDF  9  Assembly  Code. 

(2)  It  should  not  place  any  additional   burden  on  the  operators. 

(3)  It  should  record  ewery  attempt  at  an  exercise,  with  suf- 
ficient data  for  calculating  a  mark. 

(4)  It  should  provide  summaries  on  request  for  specified  classes 
and  exercises  over  a  given  period."    f7~] 

The  method  of  grading  in  the  BAGS  system  is   unique.     The  assignment  of 

credit  for  an  exercise  not  only  takes  into  account  the  mark  for  each 

program  attempted  but  also  the  number  of  attempts  to  solve  the  problem. 

The  maximum  mark  for  a  single  exercise  was  set  at  five.     One  point  was 

credited  for  each  of  the  following: 

a.  The  program  compiled  successfully. 

b.  The  program  ran  to  completion. 

c.  The  first  answer  was  correct. 

d.  The  second  answer  was   correct. 

e.  The  program  ran  successfully  within  a  prescribed  central 
processor  time. 

The  graded  results  were  printed  for  each  student  in  the  form  of  i/j. 
This  grade  states   that  the  student  made  i   attempts   at  the  exercise 
and  that  his  best  mark  was  j.      If  the  student's  program  satisfied 
all   five  criteria  the  first  time,  a  mark  of  1/5  was  assigned.     A 
grade  was  assigned  by  the  formula: 
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GRADE  =  average  of  100  x  j 

4  +  i 


The  average  was  taken  over  all  exercises  being  marked 

The  original  version  of  the  Brown  University  Student  Operating  Sys- 
tem included  a  grading  system.  The  SOS  grader  has  subsequently  been 
eliminated  from  the  SOS  system  at  Brown  University.   In  a  search  for 
properties  of  a  grader  it  is  revealing  to  analyze  the  characteristics 
of  the  original  SOS  grader  to  determine  the  cause  of  its  demise.  The 
operation  of  the  original  SOS  grading  system  is  described  as  follows: 

"Using  appropriate  supervisor  calls,  the  student  communicates 
answers  and  messages  to  the  grader  to  indicate  the  problem  to 
be  graded  (initialization),  the  answers  and  error  exits  to  be 
generated,  and  to  obtain  special  information  dependent  on  the 
specific  program  being  graded.   In  addition  to  communicating 
with  the  grader  while  his  program  is  being  executed,  the  student 
must  write  his  program  following  certain  conventions:  1)  various 
sections  of  his  program  must  bear  standard  labels  as  indicated 
in  each  problem  assignment;  2)  the  data  is  read  in  the  usual 
manner,  but  must  be  placed  in  an  assigned  location;  and  3)  the 
expected  answer  must  be  in  the  location  and  form  as  specified 
in  each  problem  assignment."  [~10l 

As  can  be  seen  by  this  description  a  great  deal  is  required  of  the 
student  to  ensure  that  his  program  is  processed  properly  by  the 
grader.  After  attempting  two  different  versions  of  the  grader  at 
Brown  University,  it  was  found  that  the  student  programs  were  re- 
quired to  initiate  almost  all  communication  with  the  grading  program 
and  doing  this  made  the  student  programs  both  more  difficult  to 
write  and  more  difficult  to  make  efficient,  due  to  the  rigidity  of 
the  communication  conventions,  A  second,  and  fatal,  limitation  of 
this  system  was  the  inability  to  specify  a  programming  problem  ex- 
plictly  enough  to  allow  the  student  to  follow  grader  conventions 
without  actually  specifying  the  method  of  solution.  A  review  of 
two  student  programs  which  had  been  graded  by  this  system  and 
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discussions  with  the  authors  of  the  system  verified  these  problem  areas 
as  the  reasons  for  elimination  of  the  grading  feature  from  the  Student 
Operating  System  at  Brown  University. 

It  is  evident  that  significant  improvements  in  automatic  grading 
techniques  have  evolved  since  the  development  of  the  Rensselaer  grader 
in  1959.     In  the  previous  discussion  of  work  in  this  field  an  attempt 
has  been  made  to  chronologically  illustrate  this  evolution.     Reference 
7  states   that  1500  first-year  students  will   be  enrolled  in  program- 
ming classes  at  the  University  of  Sydney  in  the  next  two  terms.     Under 
these  circumstances,  the  value  of  grading  systems  seems  well  establish- 
ed.    However,  much  of  the  documentation  on  this  subject  has  concerned 
implementation  of  a  particular  grading  program.     The  next  section 
proposes  certain  metrics  of  an  algorithm  which  might  be  used  in  a  grading 
system.     The  effectuation  of  some  of  these  ideas  may  not  be  practical 
at  present.     Nevertheless,  with  the  increased  interest  in  automatic 
grading  an  exploration  of  conceivable  characteristics  of  a  grader  is 
deemed  necessary. 

B.     THE  METRICS  OF  AN  ALGORITHM 

The  term  Grader  Program  is   perhaps  a  misnomer;  not  all   such  pro- 
grams assign  a  mark  or  grade  to  the  work  submitted.     A  grading  system 
is  sufficient  if  the  program  can  provide  enough  information  for  a 
mark  to  be  quickly  assigned  by  the  student's  supervisor.     The  basis 
for  the  output  of  this  information  is  discussed  in  the  subsequent 
paragraphs. 

It  is   advisable  to  discuss   the  upper  and  lower  bounds  of  a  grading 
system's  performance.     For  instance,  it  is  a  well   known  fact  that 
there  is  no  algorithm  to  determine  whether  or  not  a  given  procedure 
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is  total.  That  is,  a  grader  program  cannot  determine  whether  another 
program  contains  no  loops.  An  upper  limit  in  the  performance  of  a 
grader  program  might  be  the  class  of  functions  which  fall  in  this 
category.  A  second  class  of  characteristics  of  a  grader  which  may 
limit  achievable  performance,  is  the  class  of  functions  which  are 
computable  but  are  impractical  to  implement.  Berry  4  has  shown  that 
considerable  complication  results  in  grading  a  problem  in  which  any 
one  of  several  methods  of  solution  might  be  acceptable.  A  lower 
bound  in  the  performance  of  a  grader  is  simply  that  it  must  pro- 
duce some  information  about  the  algorithm  being  tested.  A  program 
which  recorded  the  trivial  fact  that  a  student  program  had  been 
processed  provides  information  which  might  be  used  to  evaluate  a 
student's  progress  even  though  the  program  itself  is  not  evaluated. 
Therefore,  it  can  be  concluded  that  there  exists  a  range  in  which  a 
grader  must  operate  to  produce  the  basis  for  the  evaluation  of  an 
algorithm. 

The  most  common  measure  used  in  the  evaluation  of  a  student 
exercise  is  the  determination  of  whether  or  not  the  solution  is 
correct.  This  measure  may  provide  a  sufficient  basis  for  a  grade 
for  introductory  programming  exercises  1  .  However,  it  is  sug- 
gested that  this  is  dependent  on  the  characteristics  of  the  prob- 
lem assigned.  A  grader  should  be  capable  of  providing  information 
pertaining  to  the  accuracy  of  a  program  result.   In  addition,  a 
grading  program  should  be  capable  of  measuring  this  value  in  a 
variety  of  formats  such  as  in  the  BAGS  grading  system  7  1. 

Criteria  for  evaluating  a  program  which  have  been  used  are 
successful  assembly,  successful  execution,  and  the  number  of 
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attempts  at  an  exercisel  7  .  Such  criteria  form  an  absolute  judgement 
of  a  program  in  that  the  evaluating  process  does  not  consider  the  qual- 
ity of  the  program. 

A  grader  which  only  evaluates  the  results  of  another  program  does 
not  provide  specific  information  about  the  program  itself.  Qualities 
of  a  program  which  have  significance  with  respect  to  grading  are 
execution  time,  storage  utilization,  programming  techniques,  pro- 
gram logic  and  annotation.  These  characteristics  form  a  basis  for 
the  measure  of  the  quality  of  a  program. 

The  completion  of  a  programming  exercise  within  a  sufficiently 
short  central  processor  time  was  one  criteria  in  the  BAGS  method  of 
grading  I  7j.  This  measure  of  a  program  can  provide  the  instructor 
with  a  means  of  evaluating  program  efficiency.  Application  of  this 
measurement  is  more  appropriate  for  advanced  programming  classes. 

The  amount  of  storage  utilized  by  a  program  is  a  characteristic 
which  can  be  used  to  evaluate  its  quality.  This  measure  is  all  too 
often  neglected  in  programming  classes  but  has  direct  application  in 
the  field.   Incorporation  of  this  measurement  in  a  grading  system  can 
be  an  effective  means  of  stressing  the  importance  of  the  efficient 
use  of  storage. 

If  an  instructor  compared  a  program  which  contained  one  hundred 
ADD  instructions  to  sum  one  hundred  numbers  with  a  program  which  used 
a  single  indexed  ADD  instruction  in  a  program  loop,  the  latter  would 
undoubtedly  be  considered  superior.  Although  programming  techniques 
vary  considerably,  the  quality  measure  of  a  program  may  provide  the 
best  criteria  for  evaluation  of  a  programmer's  efforts.  Due  to  the 
fact  that  good  programming  techniques  decrease  execution  time  and 
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provide  efficient  use  of  storage,  this  measure  is  more  effective  than 
analysis  of  time  or  storage  parameters  alone.  The  multiplicity  of 
thought  forms  a  substantial  barrier  to  the  implementation  of  this 
characteristic  in  a  grading  system.  A  specific  problem  could  possibly 
be  designed  which  then  would  be  inspected  for  key  variables  in  a  pre- 
scribed optimum  order  by  a  grading  system.   It  appears  that  the  im- 
plementation of  such  a  system  would  require  the  development  of  an 
extremely  capable  heuristic  pattern  matching  algorithm.   It  is  doubt- 
ful whether  the  effort  would  be  justified  in  obtaining  this  measure- 
ment. 

A  definition  of  program  logic  is  given  by  Montalbano  8  .  The 
subject  matter  of  program  logic  is  the  matching  of  appropriate  actions 
to  given  conditions.   For  example,  program  logic  is  concerned  with 
such  statements  as  "if  a  given  record  is  an  inventory  adjustment, 
process  the  record  before  all  other  records  affecting  this  stock 
item;  if  the  record  is  a  receipt,  process  it  before  any  sales  order 
affecting  this  stock  item,"  In  essence,  program  logic  is  the  ef- 
fective description  of  computer  outputs  as  functions  of  inputs. 
Techniques  such  as  narrative  description,  program  comments,  flow 
charts,  decision  tables  and  system  specification  languages  of  the 
Iverson  type  are  tools  for  expressing  program  logic  8  . 

The  logical  structure  of  an  algorithm  is  another  measure  which 
forms  a  basis  for  the  evaluation  of  the  program.  The  evaluation  of 
the  logic  of  the  program  is  extremely  difficult  because  of  the 
variety  of  paths  which  a  programmer  may  take  to  solve  the  problem. 
Chapter  IV  describes  the  implementation  of  a  grader  which  can 
evaluate  subsections  of  a  program  independently.  The  logical 


order  in  which  a  solution  was  attempted  might  be  measured  in  the  broadest 
sense  by  verifying  the  result  of  each  sequential  section  of  a  program.     A 
method  to  compare  program  logic  descriptions  such  as   those  described  in 
[8j  would  be  an  invaluable  grading  technique.     At  the  present  time  an 
inspection  by  the  instructor  of  program  logic  descriptions  such  as  flow 
charts  is   required. 

Higher  level   languages  have  an  advantage  over  assembler  languages  in 
that  they  possess  a  "self-documenting"  feature  [9 J.     The  documentation 
of  assembly  language  programs  is  an  important  area  which  requires 
evaluation.     The  annotation  of  a  program  varies   from  individual    to  in- 
dividual.    The  basic  rules  of  documentation  for  assembly  language  pro- 
grams are  described  by  Opler  I  9J.     It  is  doubtful   if  a  machine  process 
could  automatically  produce  an  evaluation  of  a  student  program  with 
respect  to  these  rules.     This  author  believes  that  graphic  methods 
may  prove  to  be  an  effective  way  of  quickly  evaluating  the  logic  and 
the  documentation  of  a  program.     A  correct  logic  flow  chart  might  be 
superimposed  on  a  student  flow  chart  on  a  display  device.     Key  decision 
blocks  could  then  be  checked  by  the  instructor.     Spot  checking  of 
annotation  can  be  accomplished  on  a  graphic  display  device  if  the  stu- 
dent programs  are  disk  resident  such  as  in  the  SOS  system  (Appendix  A). 

The  concept  of  providing  automatic  methods  which  will   evaluate  the 
quality  of  a  program  is  in  its  infancy.      It  is  expected  that  further 
work  in  this  field  will   incorporate  some  of  the  previously  listed 
metrics  of  a  program. 
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C.      GRADER  DESIGN   CONSIDERATIONS 

The  first  consideration  in  designing  a  grading  system  should  be  the 
specification  of  objectives-     What  type  of  information  is  necessary  to 
evaluate  the  programs  in  a  particular  course?     The  designer  must  con- 
cern himself  with  the  previously  mentioned  parameters  of  a  program 
which  are  required  to  be  measured. 

The  second  step  in  the  design  of  a  grader  is   to  carefully  investi- 
gate the  system  available  for  grader  use.      Forsythe     3     reports   that 
the  Stanford  grader  was  possible  only  because  of  the  well-designed 
BALGOL  compiler  with  its  own  compi ler-wi th-library  generator,  and  be- 
cause a  procedure  called  BUTTERFLY  could  generate  relocatable  machine- 
language  programs.      It  was   found  that  neither  the   IBM  7090  BALGOL 
system  nor  the  Burroughs  B5000  ALGOL  60  system  were  well   adapted  to 
the  grading  problem  at  Stanford  University. 

One  of  the  most  critical   characteristics  in  the  design  of  a  grading 
system  is  the  amount  of  involvement  of  the  student  with  the  grader  it- 
self.    The  student  should  not  be  required  to  alter  his  program  signi- 
ficantly to  conform  to  grader  conventions.     A  minimum  of  communications 
between  the  grader  and  the  student,  with  maximum  communication  capa- 
bility between  the  instructor  and  student  should  be  the  rule  in  a 
grading  operation. 

Another  feature  of  a  grading  system  which  must  be  considered  is 
the  work  required  by  the  instructor  to  program  the  grader  to  process 
the  specific  problem.     The  grading  system  should  include  macro- 
instructions   for  instructor  messages  and  general   communication 
sections   for  the  passing  of  answers   and  data.      In   general,  instructor 
involvement  with   the  grader  should  be  held  to  a  minimum. 
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The  student  should  be  able  to  work  out  solutions  using  his  own  data, 
but  data  should  be  provided  by  the  instructor  in  some  manner  at  grading 
time.  Grading  of  the  program  with  data  from  the  instructor  could  pre- 
sent situations  which  had  not  occurred  to  the  student. 

A  grading  system  should  be  able  to  handle  a  variety  of  problems. 
Programming  a  complete  grading  system  to  evaluate  a  specific  problem 
would  be  inefficient.  Manual  intervention  by  computer  operators  to 
process  grading  jobs  is  impractical  on  present  large  scale  computing 
systems  such  as  the  IBM/360.  Execution  time  of  the  student  program 
should  not  be  significantly  increased  when  executed  for  grading. 
Additional  characteristics  such  as  the  ability  to  evaluate  programs 
written  in  different  languages,  and  the  capability  to  change  problem 
parameters  with  minimum  effort  should  be  considered  in  the  design  of 
a  grading  system. 

A  summary  of  proposed  grader  design  characteristics  is  presented 
in  Table  I.  This  list  is  a  summary  of  features  which  might  be  con- 
sidered in  the  design  of  a  grading  system. 

These  characteristics  of  a  grading  system  have  been  proposed  as 
the  result  of  study  of  documented  grading  systems  and  also  from 
limited  experience  gained  in  implementing  a  practical  grader.  Such 
features,  if  only  achieved  to  some  minimum  degree,  will  provide  a 
grading  system  which  can  assist  the  instructor  greatly  in  programming 
instruction  classes.  The  next  chapter  discussed  the  mechanization  of 
a  grading  system  which  possesses  most  of  these  characteristics  and 
is  primarily  designed  for  use  in  beginning  assembly  language  pro- 
gramming courses. 
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TABLE  I 
SUMMARY  OF  GRADER  DESIGN  CHARACTERISTICS 


A.   EVALUATION  CAPABILITY 
CHARACTERISTIC  EVALUATED 


TYPE  OF  MEASURE 


Result  of  Algorithm  Absolute 

Tolerance  of  Solution  Absolute 

Successful  Assembly  Absolute 

Successful  Execution  Absolute 

Number  of  Attempts  at  Exercise  Absolute 

Assembly  Time  Quality 

Execution  Time  Quality 

Number  of  Instructions  Executed  Quality 

Storage  Utilization  Quality 

Program  Logic  Quality 

Method  of  Solution  Quality 

Documentation  Quality 

B.  MINIMUM  STUDENT  COMMUNICATIONS  WITH  GRADER 

C.  MINIMUM  INSTRUCTOR  EFFORT  IN  USING  GRADER 

D.  CAPABILITY  TO  GRADE  DIFFERENT  LANGUAGES 

E.  CAPABILITY  TO  GRADE  A  VARIETY  OF  PROBLEMS 

F.  AMOUNT  OF  OPERATOR  ATTENTION  REQUIRED 

G.  EFFORT  REQUIRED  TO  ALTER  OR  ADD  PROBLEMS  TO  GRADING  SYSTEM 
H.   DATA  MANIPULATION  FOR  PROBLEM  DURING  GRADING  RUN 
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III.   IMPLEMENTATION  OF  A  GRADER 

A.   PURPOSE  OF  GRADER 

Since  the  Student  Operating  System  (Appendix  A)  is  well -suited  for 
beginning  assembly  language  programmers,  a  grading  program  for  use  in 
introductory  programming  classes  was  written.  The  primary  purpose  of 
this  program  is  to  check  student  answers  and  assign  a  grade.  It  is 
designed  to  be  used  for  grading  small  projects  assigned  to  students 
who  are  just  learning  the  language.  With  a  grading  system  to  handle 
numerous  small  projects,  concentration  on  one  particular  area  such  as 
indexing,  looping  or  arithmetic  operations  is  possible  and  a  project 
can  be  assigned  to  cover  that  subject.  This  type  of  grader  allows 
the  instructor  to  assign  more  projects  and  gives  the  student  an  op- 
portunity to  obtain  machine  experience  on  a  wider  variety  of  assembly 
language  programming  aspects.  This  would  not  be  permissable  with  a 
system  which  was  designed  for  a  specific  problem  of  a  more  complex 
nature. 

The  grader  is  an  absolute  type  of  system  in  that  it  operates  on 
answers  from  the  student.   In  addition  to  checking  answers,  com- 
munications from  the  instructor  to  the  student  have  been  included  as 
an  essential  part  in  the  design  of  this  grader.  In  summary,  the 
purpose  of  the  grader  is  to  allow  the  instructor  to  assign  numerous 
small  projects  in  an  introductory  assembly  language  course,  to  grade 
the  projects  and  provide  messages  from  the  instructor  to  the  student. 
By  using  the  computer  as  a  tool  for  evaluation  of  class  work,  the 
instructor  can  provide  an  equal  amount  of  attention  to  appraising 
each  student's  progress. 
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B.      MECHANIZATION  OF   THE   GRADER 

Although   the  Brown  University  Student  Operating  System  Grader  pro- 
gram has   been  eliminated  from  the  system,  the  structure  for  including 
a  grader  was   intact  in  the  system  received.     By  choosing  the  GRADER= 
YES  option  at  system  generation   time,  communication   links,  control 
blocks,  and  initialization  of  parameters   for  a  grader  were  generated 
within  the  SOS  system.     Through  the  use  of  parts  of  this  available 
structure  and  modifications  to  o.    elimination  of  some  of  its   features, 
it  was  possible  to  concentrate  efforts  on  the  grading  program  itself. 

The  grader  operates  under  control   of  the  SOS  control   program,  and 
is  called  by  the  student  when  he  feels  that  his  program  is   correct. 
The  basic  structure  of  the  grading  program  and  its   interrelation  with 
the  rest  of  the  SOS  system  is  shown  in  Chart  Number  1,  Appendix  C. 
During  initialization  of  a  student's  program  the  SOS  control   program 
calls   the  bookkeeper.     The  bookkeeper  sets  system  options  specified 
by  the  user.      If  the  student  has  specified  GRADER=YES  on  his  job 
card,  grader  parameters  are  initialized  by  the  bookkeeper  and  the 
grader  program  is   called. 

A  heading  indicating  that  this   is   a  grader  run  is  printed  and 
the  SOS  interpreter  is   then  called  from  the  grader.     The  SOS  in- 
terpreter executes  SOS  machine  code  interpret!* vely.     The  first  two 
instructions  of  the  student's  program  are  executed  setting  up 
initialization  of  the  grader  for  the  particular  problem  specified 
by   the  student. 

After  grader  initialization,  the  interpreter  continues  execu- 
tion of  the  student's  program  until   it  is   desired  to  have  an 
answer  checked.     The  student   issues   a  supervisor  call   and  indicates 
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the  location  of  his  answer.  The  main  section  of  the  grader  checks  the 
answer  and  assigns  a  grade,  substitutes  the  correct  answer  if  the  in- 
structor so  desires,  prints  the  results  and  any  messages  from  the 
instructor  if  the  answer  was  not  correct.  Control  is  passed  to  the 
interpreter  again  for  further  execution  of  the  student's  program. 
This  technique  of  grading  the  student's  program  during  execution  con- 
tinues until  the  final  answer  is  graded.  Return  to  the  control  program 
is  then  made  and  the  next  student  job  is  processed. 

The  grading  program  has  an  initialization  section  to  initialize  the 
grader  for  a  specific  problem.  The  main  section  of  the  grader  handles 
all  aspects  of  grading  the  problem.  A  communications  section  for  pas- 
sing answers  and  messages  from  the  instructor  is  included.  The  grader 
is  designed  to  grade  up  to  five  intermediate  answers  for  a  problem  and 
a  final  answer.  The  capability  of  grading  twelve  different  problems  is 
included  in  the  grading  program.  Six  different  answers  should  suffice 
as  a  maximum  for  beginning  programming  projects.  The  capability  to 
grade  twelve  different  problems  allows  the  instructor  to  assign  one 
problem  per  week  during  the  class  quarter. 

The  first  part  of  the  grading  program  establishes  addressability 
with  the  other  major  sections  of  the  SOS  system  and  saves  the  return 
address  to  allow  return  to  the  control  program  at  the  end  of  the 
grading  run.  A  grader  heading  is  printed  and  the  interpreter  is 
given  control.  Exeucution  of  the  first  instruction  in  the  student's 
program  starts  the  initialization  of  the  grader.  The  return  address 
to  the  interpreter  is  first  saved  and  the  program  control  counter  is 
incremented  to  point  to  the  next  sequential  instruction  in  the  stu- 
dent program.  A  check  for  student  core  wrap  around  is  made  and  a 
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check  for  a  proper  initialization  call  is  accomplished.  A  halfword 
grader  control  block  is  used  throughout  the  program  to  set  various  con- 
ditions. The  first  byte  of  this  control  block  is  used  by  the  system. 
The  bookkeeper  sets  a  bit  on  if  the  student  has  asked  for  a  grading 
run.  The  grader  initialization  section  turns  a  second  switch  on  if 
grading  is  in  process.  The  second  byte  of  the  grader  control  block 
is  used  by  the  instructor  to  control  various  functions  of  the  main 
grading  section  and  its  use  will  be  described  in  the  section  on  use 
of  the  grader. 

If  a  grading  run  has  been  requested  and  initialization  has  not 
yet  taken  place,  the  data  for  the  problem  specified  is  placed  in  a 
general  communication  section.  This  is  accomplished  by  the  use  of  a 
dummy  section  which  describes  the  layout  of  storage  for  the  data  that 
the  instructor  has  defined  for  a  specific  problem.  Symbolic  names 
defined  in  the  dummy  section  are  then  used  as  operands  in  the  main 
grading  section  during  the  grading  process,  This  method  allows  the 
instructor  to  separately  assemble  the  appropriate  data  in  a  control 
section  and  use  the  link  editor  to  link  his  answer  CSECT  in  object 
form  to  the  system. 

Various  error  routines  and  messages  are  included  in  the  grader 
initialization  section.  Upon  completion  of  initialization,  a  mes- 
sage to  that  effect  is  printed  and  execution  of  the  student's 
program  is  continued.   The  grading  program  grades  the  result  of  up 
to  five  intermediate  sections  which  the  student  has  used  in  his 
program  to  achieve  a  final  answer  to  a  specific  problem.  When  the 
student  has  reached  a  particular  result,  the  main  grader  section 
of  the  grading  program  is  called.   This  section  first  determines  which 
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answer  has  been  specified,  and  then  branches  to  the  appropriate  routine 
The  grading  routines  for  intermediate  answers  are  similar  except  for 
correct  answer  usage  data  and  the  messages  sent  to  the  student. 

The  intermediate  answer  grading  routines  first  obtain  the  student's 
answer.  This  is  accomplished  by  the  subroutine  called  GETANS.  The 
address  of  the  answer  in  student  core,  which  is  word-oriented,  is 
obtained  and  converted  to  a  System/360  address.  The  correct  answer 
specified  by  the  instructor  is  then  compared  with  the  student's 
answer.  If  the  answer  is  correct,  a  message  indicating  this  fact  is 
printed  and  credit  for  that  answer  is  accumulated.  The  accumulated 
credit  for  each  intermediate  answer  is  also  printed.  If  the  student 
answer  is  not  correct,  a  test  is  made  to  see  if  the  instructor  has 
a  message  for  the  student  which  might  assist  him  in  obtaining  the 
correct  answer.  If  the  instructor  has  included  a  message,  it  is 
printed  on  the  student's  output.  The  instructor  has  the  option  of 
substituting  correct  data  for  a  wrong  answer  if  he  so  desires.  Thus 
the  subsections  of  a  student  program  can  be  checked  even  if  the  pro- 
gram as  a  whole  computes  incorrectly.  If  the  instructor  has  chosen 
this  option,  the  correct  data  is  moved  into  the  student  core  address 
of  his  incorrect  answer.  No  credit  is  accumulated  for  a  wrong 
answer.  A  message  indicating  that  the  answer  was  not  correct  is 
printed. 

If  there  is  just  one  answer  for  a  particular  problem  or  if  a 
final  solution  has  been  reached,  the  final  answer  routine  of  the 
main  grading  section  is  entered.  The  final  answer  routine  checks 
the  solution  and  prints  a  message  indicating  whether  it  is  correct 
or  not.  A  total  grade  is  then  determined.  Fifty  points  are  given 
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for  a  correct  final  answer   Each  of  the  five  intermediate  answers  are 
worth  ten  points.  A  grade  for  a  problem  with  five  intermediate  answers 
and  a  final  answer  is  totaled  by  summing  the  accumulated  intermediate 
answer  credit  and  the  credit  for  the  final  answer.   If  the  problem  re- 
quired less  than  five  intermediate  answers,  the  accumulated  credit  for 
the  intermediate  answers  is  first  supplemented  to  fifty  and  then  added 
to  the  credit  for  the  final  answer.  The  assignment  of  these  credit 
values  for  intermediate  and  final  answers  is  arbitrary  and  may  be 
changed  by  an  instructor  with  only  minor  modifications  to  the  grading 
program.  The  total  machine  grade  and  any  final  message  to  the  student 
is  then  printed  on  the  student's  output. 

The  next  part  of  the  grading  program  evaluates  the  efficiency  of 
the  student's  program  to  some  extent.  The  number  of  instructions 
executed  in  the  student  program  is  printed:  The  mean  number  of  in- 
structions executed  in  previous  programs  for  this  project  is  supplied 
by  the  instructor  in  his  answer  control  section.  These  two  values 
are  compared  and  additional  credit  is  given  to  the  student  whose  pro- 
gram required  the  execution  of  fewer  instructionSc  A  message  indi- 
cating whether  the  student  required  more  or  less  than  the  median  is 
printed.  The  total  machine  grade  with  bonus  credit  is  then  printed. 
This  capability  is  optional  and  is  set  by  the  instructor  in  the 
answer  control  section 

A  return  to  the  SOS  control  program  is  then  made  for  further 
processing  of  student  jobs   The  grader  program  is  included  in 
Appendix  B.   Flowchart  documentation  of  the  program  is  included  in 
Appendix  C< 
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C.      USE  OF  THE  GRADER 

This  section  describes  the  conventions  which  must  be  followed  by  the 
student  and  the  instructor  to  ensure  proper  operation  of  the  grading 
system.     Examples  of  graded  student  programs  are  shown  in  Appendix  D. 
The  control  section  established  by  the  instructor  to  communicate 
answers  and  messages  to  the  grader  is  also  included  as  Appendix  E. 
1.     Student  Grading  Conventions 

The  student  is  required  to  indicate  four  items  to  the  grader. 
He  must  first  specify  that  he  desires  his  program  to  be  graded.     The 
student  then  indicates  which  problem  is  to  be  graded.     When  the  stu- 
dent has  reached  a  solution  to  the  problem  he  must  indicate  to  the 
grader  which  of  the  six  possible  answers  he  wants  to  have  checked  and 
the  location  of  that  answer. 

a.     Calling  for  a  Grading  Run 

When  the  student  feels  that  his  program  is  ready  for  grading, 
he  utilizes  the  JCL  bookkeeping  parameter  option  feature  of  the  SOS 
system  to  call   the  grader.     This  is  done  by  placing  GRADER=YES  on  the 
job  card  starting  in  column  26. 
EXAMPLE: 


Column:  16  16  26 


/JOB   BACONRF    S0SJ0B1    GRADER=YES 


b.  Specification  of  the  Problem  to  be  Graded 

To  indicate  which  problem  is  to  be  graded  the  student 
places  as  the  first  executable  instruction  of  his  program  the  super- 
visor call  SVC  3,0  and  a  second  instruction  DC  (decimal  number  0 
through  11)  immediately  following  the  supervisor  call.  The  super- 
visor call  initializes  the  grader  for  the  problem  specified  in  the 

29 


second  instruction   For  example,  if  the  student  desired  problem  number 
three  to  be  graded,  the  first  two  instructions  of  his  program  would  be 
as  follows: 


cr  Specification  of  the  Answer  to  be  Graded 

The  student  must  indicate  to  the  grader  which  answer  is 
to  be  graded.  This  is  accomplished  by  issuing  a  supervisor  call. 
EXAMPLE: 


SUPERVISOR 

CALL 

SPECIFIES  THIS  ANSWER 
First  intermediate  answer 

SVC 

3,1 

SVC 

3,2 

Second  intermediate  answer 

SVC 

3,3 

Third  intermediate  answer 

SVC 

3,4 

Fourth  intermediate  answer 

SVC 

3,5 

Fifth  intermediate  answer 

SVC 

3,6 

Final  Answer 

d.  Specifying  the  Location  of  the  Answer 

Immediately  following  the  supervisor  call  specifying 
the  answer,  the  student  places  a  HALT  instruction  with  a  single 
operand  which  is  the  symbolic  name  of  the  location  of  the  answer. 
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An  example  of  a  student  requesting  a  check  of  his  fourth  intermediate 
answer  which  had  been  previously  stored  in  a  storage  location  called 
ANSWER  would  be  as  follows: 


Column:  10       16 


SVC      3,4 
H        ANSWER 


2.  Instructor  Grading  Conventions 

The  instructor  must  indicate  four  items  to  the  grading  system: 
the  number  of  answers  to  be  graded;  the  correct  answers;  messages  to 
the  student;  and  grader  options. 

a.  Specification  of  Answers 

To  pass  correct  answers  to  the  grader,  the  instructor  de- 
fines fullword  constants  through  the  use  of  a  macro-instruction  called 
ANS.  The  answers  will  be  defined  in  the  instructor's  control  section. 

EXAMPLE: 


Column:  10      16 


ANS     8,13,-99,33397 


The  above  example  defines  three  intermediate  answers  and  a 
final  answer.  Answers  must  be  specified  in  order. 

b.  Specification  of  the  Number  of  Answers 

The  number  of  answers  to  be  graded  is  defined  as  a  full 
word  constant  in  the  control  section. 
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EXAMPLE 


Co  1 umn : 

1 

10 

16 

NOINTANS 

DC 

F ' decimal    number  0-5' 

c.     Specification  of  the  number  of  instructions  executed. 

The  instructor  specifies  a  value  for  the  grader  to  use  in 
the  comparison  of  instructions  executed  by   the  student's  program  by 
defining  a  fullword  constant  in  the  control   section. 

EXAMPLE: 


Column:  1  10  16 


NOX  DC  F1 decimal   number1 


d.     Specification  of  grader  options 

The  grading  system  has  three  optional   features.     Messages 
to  the  student,   the  substitution  of  correct  answers,  and  the  assign- 
ment of  additional   credit  are  chosen  by  the  instructor  through  the 
use  of  a  macro-instruction  called  PROF.     If  it  is   desired  to  include 
all    three  features   the  default  is   chosen  as   follows: 


Column:  10  16 


PROF       (blank  operand  field) 


Operands  to  eliminate  these  features  are: 

NOMSG  -  eliminates  message  feature 
NODATA  -  correct  answers  ^re   not  substituted 
N0B0NUS-  feature  to  check  the  number  of  instructions 
executed  and  asbign  bonus  credit  is  removed 
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EXAMPLE:  To  prevent  the  substitution  of  correct  answers  and 
eliminate  the  assignment  of  additional  credit  the 
instructor  writes: 


Column:  10        16 


PROF      NODATA,  NOBONUS 


Note:  The  operands  may  be  placed  in  any  order  in  the  operand  field. 

e.  Specification  of  messages 

The  instructor  may  send  a  message  to  the  student  whenever  an 
incorrect  answer  is  obtained  and  at  the  end  of  a  grading  run.  These 
grader  messages  are  defined  through  the  use  of  a  macro-instruction  called 
IMSG.  The  instructor  may  define  as  many  as  six  messages  in  the  control 
section.  The  labels  for  the  messages  are  IMESS1  through  IMESS5  for 
intermediate  answer  messages  and  IMESSF  for  a  final  message.  Messages 
can  be  up  to  132  characters  in  length. 

EXAMPLE: 


Column: 

1 

10 

16 

IMESS4 

IMSG 

'message  to  student' 

Note:     This  message  would  be  printed  if  the  student's  fourth 
intermediate  answer  were  incorrect. 

The  order  in  which  these  items  must  be  defined  in  the  con- 
trol  section  is  shown  in  the  example  in  Appendix  E.     The  instructor 
specifies  these  parameters  in  a  control   section  with  labels  which 
indicate  the  particular  problem.     GINITO  indicates   the  first  problem 
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whereas  GINIT11  indicates  the  twelfth  problem.  The  control  section  can 
be  assembled  and  an  object  deck  obtained  using  the  System/360  ASMAD  pro- 
cedure. The  control  section  may  then  be  linked  to  the  SOS  system  by  the 
linkage  editor. 

D.   FEATURES  AND  LIMITATIONS  OF  THE  GRADER 

A  comparison  of  the  characteristics  of  this  grader  with  the  general 
properties  of  grading  systems  listed  in  the  previous  chapter  is  now 
presented.  The  SOS  grader  accomplishes  the  basic  objective  of  de- 
termining whether  an  answer  is  correct  or  not  and  assigning  a  grade. 

The  student  involvement  with  the  grading  system  itself  has  been 
kept  to  a  minimum.  Very  little  is  required  of  the  student  to  indicate 
the  problem  to  be  graded  and  correct  answers. 

The  student  can  be  assisted  in  his  programming  by  messages  from 
the  instructor.  This  feature  allows  up  to  six  messages  from  the  in- 
structor to  the  student.  These  messages  are  predefined  to  fit  the 
circumstances  of  the  problem  solution. 

Through  the  use  of  a  general  communications  section,  ease  of 
specifying  the  answers  and  messages  for  various  problems  is  made 
possible.  Macro-instructions  for  use  with  the  answer  control  section 
allow  the  instructor  to  provide  grader  data  with  minimum  effort. 

A  feature  of  the  grader  is  its  flexibility  in  grading  as  many  as 
six  answers  for  one  problem  and  its  ability  to  grade  twelve  dif- 
ferent problems  during  one  batch  run  of  student  jobs.  Students  may 
work  on  several  projects  at  the  same  time  because  of  this  capability. 

The  substitution  of  correct  intermediate  answers  allows  indepen- 
dent grading  of  the  subsections  of  the  student's  program.  By 
evaluating  each  section  of  a  program  separately,  a  more  representa- 
tive grade  can  be  assigned. 

34 


The  only  feature  of  the  grader  which  examines  the  quality  of  a  stu- 
dent's program  is  the  examination  of  the  number  of  instructions  executed, 
A  better  measurement  of  the  efficiency  of  the  program  would  be  the 
execution  time  of  the  student  program.     The  grader  does  not  evaluate 
storage  utilization  or  the  number  of  runs  submitted  prior  to  grading. 
Inherent  characteristics  of  the  SOS  system  can  be  used  by  the  instruc- 
tor to  assign  a  grade  based  on  these  parameters.     These  characteristics 
are  discussed  in  Appendix  A. 

The  SOS  system  uses  word-oriented  storage.     Because  of  this,  the 
grader  is  designed  to  evaluate  fullword  answers.     A  minor  limitation 
is  the  fixed  credit  structure  of  the  grader.     However,  this  can  be 
modified  to  suit  a  particular  instructor's  desires.     Since  the  grader 
is  designed  for  use  in  beginning  programming  classes,  these  limitations 
do  not  affect  the  capability  to  grade  the  type  of  program  expected  in 
introductory  class  work. 
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IV.   CONCLUSIONS  AND  RECOMMENDATIONS 

A.   CONCLUSIONS 

A  grading  system  for  use  with  the  Student  Operating  System  in  begin- 
ning assembly  language  classes  has  been  implemented  at  the  U.S.  Naval 
Postgraduate  School.  This  system  will  significantly  aid  the  instructor 
who  must  grade  student  exercises  in  classes  with  a  large  number  of 
students. 

This  type  of  grader  will  provide  the  capability  to  assign  more 
exercises  during  the  academic  quarter.  Programming  instruction  can 
emphasize  various  aspects  of  the  language  in  each  assignment,  thus 
allowing  the  student  to  gain  more  experience  in  assembly  language 
programming. 

Since  the  evaluation  of  a  programming  exercise  is  inevitably  re- 
petitive, the  concept  of  writing  a  grader  program  which  would  be  able 
to  check  other  programs  is  both  practical  and  useful.  Documentation 
of  previous  grading  systems  has  emphasized  the  implementation  of  the 
grading  program.  This  investigation  has  examined  the  overall  con- 
cept of  a  grading  system.  Characteristics  of  a  grader  have  been 
evaluated.  Several  of  these  characteristics  which  measure  the  quality 
of  a  program  would  be  extremely  difficult  to  implement  at  the  present 
time.  Despite  these  complexities,  it  is  anticipated  that  grading 
systems  of  the  future  will  concentrate  on  the  evaluation  of  the  qual- 
ity of  a  program. 

A  significant  quality  of  grading  systems  in  general  is  the  fact 

that  the  machine  may  be  more  objective  in  grading  than  the  human, 

because  of  its  notable  lack  of  prejudice  and  its  inability  to  become 

bored  3J. 
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Grading  systems  generate  a  paradoxical  situation.  A  system  which 
is  designed  to  minimize  student  involvement  with  the  grader  itself, 
invariably  requires  more  effort  on  the  part  of  the  instructor.  A  grader 
which  not  only  checks  answers  but  also  evaluates  the  quality  of  the  pro- 
gram may  require  considerable  additional  labor  by  the  instructor. 
Because  of  this  situation,  tradeoff  between  student  requirements  and 
instructor  efforts  must  be  fully  evaluated  prior  to  the  design  of  a 
grading  system. 

The  grading  system  should  allow  the  instructor  flexibility  in 
generating  class  assignments  by  minimizing  programming  work  on  his 
part  with  respect  to  the  grader.  It  should  be  recognized  however, 
that  once  the  initial  effort  is  expended  in  generating  the  problem, 
it  can  be  utilized  many  times. 

When  a  grading  system  is  to  be  programmed,  it  behooves  its  de- 
signer to  thoroughly  examine  the  system  with  which  it  will  interact. 
The  communication  capabilities  and  limitations  of  the  system  must  be 
evaluated.  The  system  parameters  such  as  execution  time,  the  number 
of  instructions  executed,  and  storage  limitations  should  be  examined 
before  the  initial  design  of  a  grading  system.  In  this  manner, 
various  existing  facilities  may  be  used  in  the  grading  system. 

B.   RECOMMENDATIONS 

It  is  strongly  recommended  that  the  grading  system  be  used  at  the 

Naval  Postgraduate  School.  It  represents  a  powerful  aid  to  assembly 

language  programming  instruction. 

Recommendations  for  augmentation  of  the  grader  are: 
(a)  Inclusion  of  the  standard  data  option  of  the  Student  Opera- 
ting System  to  be  used  in  conjunction  with  the  grader. 
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(b)  Revision  of  the  Student  Operating  System  statistics  keeper  so 
that  it  can  produce  a  grade  book  based  on  results  obtained 
from  the  grader. 

(c)  Modification  of  the  grader  by  replacing  student  supervisor 
calls  to  the  grader  with  simple  Student  Operating  System  macro- 
instructions. 

(d)  Provide  the  Student  Operating  System  with  a  meaningful  timer 
and  revise  the  grader  to  utilize  execution  time  as  a  measure 
of  the  efficiency  of  student  programs. 

(e)  Assignment  of  these  recommendations  to  students  as  suitable 
term  projects  in  an  assembly  language  programming  class. 

Very  little  work  has  been  done  in  developing  effective  techniques 
to  automatically  evaluate  the  quality  of  a  program.  It  is  recommend- 
ed that  further  investigation  of  both  the  theoretical  and  the  practi- 
cal phases  of  this  area  of  grading  be  conducted. 
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APPENDIX  A 

The  Student  Operating  System 

A.   PROGRAMMING  INSTRUCTION  AT  THE  POSTGRADUATE  SCHOOL 

Students  taking  a  beginning  Computer  Science  course  at  the  Naval 
Postgraduate  School  have  ample  opportunity  for  actual  machine  ex- 
perience. This  is  not  the  case  for  some  universities  where  cost 
considerations  take  precedence  and  research  work  usually  justifies 
the  existence  of  the  university  computer  center. 

To  add  credence  to  the  idea  of  efficiently  instructing  programming 
courses,  an  indirect  measure  of  the  amount  of  programming  instruction 
at  the  Postgraduate  School  was  obtained.  This  measure  is  the  number 
of  WATFOR  jobs  processed.  WATFOR  is  an  acronym  for  the  Waterloo  FOR- 
TRAN compiler.  This  system  has  been  used  extensively  for  introductory 
FORTRAN  programming  classes  since  October,  1967. 

Table  II  presents  the  Naval  Postgraduate  School  Computer  Facility 
usage  data.  Approximately  forty  percent  of  the  total  number  of  com- 
puting jobs  submitted  in  the  past  eighteen  months  have  been  WATFOR 
jobs.  This  is  an  indication  that  a  primary  use  of  the  Computer  Facil- 
ity is  programming  instruction.  Many  of  the  jobs  listed  in  the 
STUDENT  category  of  Table  II  are  also  programming  instruction  class 
work.  Faculty  programming  jobs  comprise  8.75  percent  of  the  total 
and  are  assumed  to  be  primarily  associated  with  research  studies. 
This  evidence  indirectly  substantiates  the  fact  that  a  major  use  of 
the  Computer  Facility  is  programming  instruction. 
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B.  THE  ORIGINAL  STUDENT  OPERATING  SYSTEM 

A  system  which  can  be  used  for  assembly  language  instruction  is  the 
Brown  University  Student  Operating  System  10 1.  This  student  operating 
system  provides  an  assembler  which  produces  code  for  a  simplified 
machine,  an  interpreter  which  simulates  the  simplified  machine  and  a 
control  program  for  storage  manipulation,  editing  of  the  student's  pro- 
gram and  statistics  gathering.  Wile  10  documents  the  capability  of 
this  system  to  provide  assembly  language  teaching  and  processing  at 
costs  comparable  to  other  university  algebraic  compilers  used  for 
student  programming  instruction. 

The  Student  Operating  System  (SOS)  offers  several  advantages  to  the 
instructor  who  wishes  to  utilize  the  IBM  System/360  in  an  assembly 
language  class.  The  feature  of  changing  basic  parameter  settings  at 
system  generation  time  or  taking  default  options  allows  the  instructor 
to  change  the  capabilities  of  the  entire  system  for  particular  pro- 
gramming projects.  All  of  the  system  is  in  core  at  all  times  and  set 
up  for  each  student  job  requires  less  than  twenty  machine  instructions. 
SOS  utilizes  a  simplified  job  control  language,  thus  practically 
eliminating  the  difficult  task  of  explaining  the  us-e  of  and  reason 
for  the  complex,  but  extremely  powerful,  general  purpose  JCL  required 
by  the  System/360  Operating  System.  Student  jobs  are  cataloged  on 
a  disk;  editing  is  done  directly  on  the  cataloged  programs,  thus 
introducing  the  student  to  the  concept  of  text  editing  and  library 
maintenance  as  well  as  reducing  the  volume  of  cards  which  the  computer 
is  required  to  read. 

The  SOS  machine  and  assembly  languages  were  designed  to  eliminate 
some  of  the  more  difficult  programming  concepts  inherent  in  the 
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System/360  machine  and  assembly  languages  10  .  However,  the  SOS  pro- 
cessing languages  provide  a  compatible  introduction  to  the  related 
System/360  languages  as  the  SOS  assembly  language  mnemonics  are,  in 
general,  a  subset  of  the  System/360  mnemonics.  Arithmetic  and  logical 
operations  are  performed  in  a  similar  manner  to  System/360  operations. 
Data  is  in  two's  complement  form  and  operations  are  performed  in 
registers  producing  results  and  interruptions  similar  to  360  instruc- 
tions. 

The  instructor  may  view  the  elimination  of  base/displacement  ad- 
dressability considerations,  variable  length  instruction  and  data 
formats,  floating  point,  packed  decimal  and  character  format  con- 
versions, relocatability,  control  and  dummy  section  capabilities, 
and  condition  code  testing  from  the  assembly  language  repertoire  as 
being  too  restrictive  for  the  proper  introduction  of  assembly  language 
programming  to  the  student.  However,  it  is  the  opinion  of  this  author 
that  provisions  included  in  the  SOS  system  for  indexing,  indirect 
addressing,  register  addressing,  overflow  testing,  and  all  arith- 
metic and  logical  operations  are  adequate  to  accomplish  the  objective 
of  assembly  language  instruction  at  this  school.  A  reasonable  state- 
ment of  this  objective  is  to  provide  the  student  with  an  under- 
standing of  the  structure  and  organization  of  a  computing  machine. 
It  is  not  the  intent  of  this  study  to  document  in  detail  the  Student 
Operating  System  and  its  language.  Specifications  for  the  system 
and  a  description  of  its  use  may  be  found  in  Ref.  11. 

C.   IMPLEMENTATION  OF  THE  SOS  SYSTEM 

The  SOS  System  was  obtained  from  Brown  University  in  October, 
1968.  Documentation  of  the  system  received  was  negligible.  System 
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generation  was  accomplished  by  issuing  a  single  macro  instruction  (SOS); 
all  program  segments  were  included  as  lower  level  macro  calls.  This 
version  was  unsuitable  for  practical  use  in  implementing  a  grader  pro- 
gram within  the  system  or  for  making  changes  to  the  system  in  general. 
The  "SOS"  macro  included  system  default  parameters.  Required  global 
set  symbols  were  included  in  this  macro.  The  main  section  of  the  "SOS" 
macro  was  the  SOS  processor  common  area.  This  section  called  the  eight 
major  macros  in  the  system.  The  system  assembled  as  the  single  macro- 
instruction  "SOS"  with  this  structure. 

The  IBM  System/360  assembly  language  includes  the  COPY  instruction. 
This  instruction  obtains  source  language  coding  from  a  library  and  in- 
cludes it  in  the  program  currently  being  assembled.  The  assembler 
inserts  the  requested  coding  immediately  after  the  COPY  instruction 
is  encountered.  This  instruction  was  utilized  to  provide  a  system 
structure  which  allowed  manipulation  of  each  of  the  eight  major 
sections  separately.  The  macro  structure  of  each  of  the  previously 
mentioned  macros  was  removed  and  all  global  set  symbols  were  placed 
in  a  section  named  SOSGBL  which  was  then  used  as  copy  text  in  the 
SOS  macro.  Keyword  parameters  in  the  SOS  macro  were  modified  to 
make  them  compatible  with  the  revised  structure.  In  addition,  the 
SOS  macro  was  shortened  to  include  only  the  necessary  set  symbol 
instructions.  The  original  SOS  processor  common  area  was  modified 
to  call  in  the  eight  major  program  sections  of  the  system  by  the 
use  of  COPY  statements  instead  of  the  previous  macro-expansion 
method.  This  section  of  coding  was  renamed  SOSMAIN.  The  modified 
version  of  the  system  was  generated  on  7  February  1969. 
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D.   SOS  CHARACTERISTICS  FOR  STUDENT  EVALUATION 

Several  inherent  characteristics  of  the  SOS  system  interrelate  with 
the  concept  of  grading  a  student's  programming  work.   In  describing  the 
operation  of  a  grader  in  use  at  Stanford  University,  Forsythe  and  Wirth 
2 J  state  that  it  is  inappropriate  to  grade  beginning  students  on  the 
execution  time  of  their  programs  or  to  evaluate  the  amount  of  storage 
used  by  their  programs.   Certainly  the  novice  programmer  should  not  be 
severely  restricted  in  his  programming  efforts  by  these  parameters. 
However,  he  should  be  made  aware  of  their  existence  and  thus  eliminate 
the  breeding  of  poor  programming  habits  at  the  outset  of  instruction. 
By  using  the  capability  of  varying  system  parameters  in  the  SOS  system 
the  instructor  may  impose  restrictions  which  will  require  the  student 
to  consider  such  aspects  of  programming  as  storage  utilization,  the 
number  of  instructions  executed,  register  usage,  and  the  number  of 
program  runs  to  achieve  a  solution. 

The  capability  of  imposing  restraints  on  the  student  is  essentially 
a  method  of  evaluating  his  ability  to  program  within  the  restricted 
environment.   It  is  not  an  absolute  grading  mechanism,  but  rather  a 
circuitous  method  of  examining  his  programming  ability.  The  features 
of  the  SOS  system  which  provide  this  capability  are  discussed  in  the 
subsequent  paragraphs  of  this  section. 

The  highest  configuration  of  SOS  core  the  instructor  wishes  his 
students  to  use  can  be  set  by  the  keyword  parameter  MAXCNF  = 

(1/2/ /8)  at  system  generation  time  or  by  the  bookkeeping  parameter 

CONF  =  (1/2/.... /8)  at  execution  time.  The  student  core  size  is  then 
set  to  the  value  specified  times  512  words.  An  instructor  could 
generate  a  problem  whose  solution  required  approximately  1000  words 
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of  storage  and  then  specify  in  the  assignment  that  the  bookkeeping  param- 
eter CONF  =  2  must  be  used  on  the  student's  job  control  card.  If  the 
student  solved  the  problem  within  a  1024  storage  environment,  he  could 
be  given  additional  credit.  The  poorer  student  could  still  achieve  the 
solution  but  would  use  the  default  parameter  of  8  (4096  words).  The 
actual  configuration  used  is  printed  on  the  first  page  of  the  student's 
output  by  the  bookkeeper.  The  ability  to  vary  student  core  size  is 
beneficial  to  the  student  because  he  can  be  made  aware  of  real -world 
machine  limitations  at  the  start  of  his  programming  experience. 

One  means  of  student  evaluation  is  to  limit  the  number  of  program- 
ming runs  that  a  student  may  submit  prior  to  final  grading  of  a  project. 
This  method  insures  that  the  student  will  carefully  think  out  each  step 
of  his  program  and  thoroughly  review  each  instruction  before  submitting 
it  for  execution. 

The  SOS  system  provides  the  capability  to  record  the  number  of  runs 
a  student  submits  for  execution.  If  the  system  is  used  properly,  stu- 
dents will  be  required  to  catalogue  their  programs  in  the  student  job 
library  on  a  disk.  All  changes  to  the  program  are  then  made  through 
the  use  of  the  SOS  editing  feature.  On  the  first  page  of  the  student's 
output  under  the  data  card  listing  a  message  is  printed  "NEXT  RUN  IS 

NO. . "  This  provides  the  instructor  with  a  means  of  evaluating  the 

ability  of  the  student  to  achieve  a  solution  to  the  problem  in  as  few 
runs  as  possible.  Again  this  characteristic  of  the  system  does  not 
pass  an  absolute  judgement  on  a  student's  solution  to  a  programming 
project.   It  does  however,  provide  a  better  means  of  evaluating  the 
overall  potential  of  the  student. 
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A  third  built-in  grading  characteristic  of  the  SOS  system  is  the 
capability  to  vary  the  maximum  number  of  allowable  executable  instruc- 
tions. The  primary  purpose  of  this  feature  is  to  prevent  infinite 
looping.  The  maximum  number  of  allowed  executable  instructions  may  be 
set  at  system  generation  time  by  the  keyword  parameter  SOSMAX  = 
(   ,   ,  maximum  number  of  instructions)  or  at  execution  time  by  the 
student  through  the  use  of  the  bookkeeping  parameter  INSC  =  (decimal 
number)  on  the  student's  job  card.  This  parameter  is  evaluated 
directly  by  the  Grader  and  is  discussed  in  Chapter  III. 

In  summary,  the  major  capabilities  of  the  SOS  system  with  respect 
to  student  program  evaluation  are  storage  utilization  flexibility,  the 
capability  to  record  every  attempt  at  an  exercise,  and  variability  of 
the  maximum  allowable  number  of  executable  instructions.  Imaginative 
and  prudent  application  of  these  features  of  the  SOS  system  by  an 
instructor  should  motivate  the  student  toward  good  programming  prac- 
tices and  at  the  same  time  allow  a  broader  evaluation  of  the  true 
capability  of  the  student. 
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APPENDIX  B 

The  Grader  Program 
The  Grader  Program  is  shown  in  assembled  form.     The  Grader  Program 
is  assembled  with  the  Student  Operating  System  when  the  keyword  param- 
eter GRADER=YES  is   chosen  at  system  generation  time. 
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APPENDIX  C 

Flowchart  Documentation  of  the  Grader  Program 

The  flowcharts  which  document  the  various  sections  of  the  Grader 
Program  are  the  following: 
Flowchart  Number  Title 

1.  The  Basic  Grading  Routine 

2.  Grader/SOS  Interface  and  Grader 

Initialization 

3.  Intermediate  Answer  Grading  Routine 

4.  Final  answer  Grading  Routine 

5.  Routine  for  Accumulating  Intermediate 

Answer  Credit 

6.  Routine  to  Obtain  Student's  Answer 

7.  Translation  and  Printout  of 

Machine  Grade  Routine 

8.  Return  Routines 

9.  Total  Machine  Grade  Routine 

(First  Section) 

10.  Total  Machine  Grade  Routine 

(Second  Section) 

11.  Instructions  Executed  Routine 

12.  Structure  of  General  Communications 

Section 
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***  APPENOIX  C  —  CHART  NUMBER  1  *»* 
FLOW  CHART  OF  THE  BASIC  GRADING  ROUTINE 


»************«» 

START  —  SOS 

CONTROL 

PROGRAM 

*************** 


*************** 

>  « 

SOS 

BOOKKEEPER 

(CALLS  THE 

•     GRAOER)     * 

****•••******•* 


*  *NC 

*  IS  THE  GRAOER  * 

IN  THE  SCS 

*  SYSTEM?     * 


************** 
• 

*  INITIALIZE 

*  GRADER 

*  PARAMETERS 
* 
•****•**•****• 


************** 


************** 


•**•*****•**•** 

*  * 
•PRINT  HEAOING* 

*  AND  CALL    *- 

*  INTERPRETER  » 

*  FRCM  GRADER  * 
*************** 


INTERPRET 

*************** 


EXECUTION 
CYCLE 


*************** 


*****»«**»***♦» 


**••***•******* 


HAS  STUDENT 

ASKED  FOR 

GRADER? 


*************** 

*  * 
♦PREPARATIONS  * 

*  FOR  MAIN    * 

*  GRADER     * 

*  * 
*************** 


*************** 

•  * 
•INSTRUCTOR'S  * 

•  ROUTINE  TC   • 

*  INITIALIZE   * 

*  GRAOER     * 
*************** 


*************** 

*  • 

*  SET  THE    * 

*  GRADER  "IN   * 

*  PROCESS"    * 

*  * 
*************** 


*************** 

*  * 

*  RETURN  TC   » 

*  INTERPRETER  * 
»  FOR  FURTHER  * 

*  EXECUTION   * 
*************** 


•**•**•**•*•*** 

EXIT  TO 
INTERPRETER 

*************** 


********•*•••** 

*  * 

*  SUBSTITUTE   * 
•CORRECT  DATA  * 

*  IF  REQUIRED  » 

*  * 
*************** 


********************* 

•      PRINT-       * 

MESSAGES  TO 

»   THE  STUDENT   * 

*************** 


********************* 

*   PRINT  CREDIT    * 

FOR  CORRECT 

*     ANSWER      * 

*•*•*•*****•*** 


*  *NO 

* 
FINAL  ANSWER? 


*************** 

*  * 

*  RETURN  FOR   * 
->»  NON-GRAOING  * 

*  RUN      » 

*  * 
*************** 


*************** 

EXIT  TO 
INTERPRETER 

*************** 


**•*•***•*•••******•* 


*************** 


*****••***•*•** 

*  * 

*  RETURN  TO   * 

*  CONTROL    * 

*  PROGRAM  VIA  * 

*  BOOKKEEPER   * 
*************** 


**•*••***•***** 

EXIT  TO 
BOOKKEEPER 

***••••***••*** 


*************** 


*************** 


*************** 

EXIT  TO 

INTERPRETER 
*************** 
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•••  APPENDIX  C  —  CHART  NUMBER  2  •*• 
GRADER/SOS  INTERFACE  AND  GRADER  INITIALIZATION 


START-  GRADER 
ENTRY/RETURN 


START- 
INITIALIZE 
GRADER 


• 

♦  ESTABLISH 
»  AODRESSING 

♦  WITH  SCS 

♦  SYSTEM 
.............. 


* 

♦CEFINE  GRADER 

*  OUTPUT 

•  HEADING 

* 


* 

*  SAVE  RETURN 

*  TO  SOS 
»  CONTROL 

*  PROGRAM 


..................... 

*   PRINT  GPACER    • 

OUTPUT 

*     HEADING     * 


••••••••••••••• 

•  • 
»  SAVE  RETURN  • 

*  TO       ♦ 

*  INTERPRETER  » 

•  * 


*  INCREMENT   * 

*  CONTROL  * 
»  COUNTER  TO  • 
♦POINT  TC  NEXT* 

*  INSTR.     • 


*  • 

*  CHECK  FCR  * 
•STUDENT  CCRE  • 

*  WRAPAROUND  * 

*  * 


* 

*  LOAD 

*  SUPFRVISCR 

*  CALL  COPE 

* 


"THIS  IS  AN 

SCS  GRADER 

RUN   GOOD 

LUCK" 


............... 

*  PRANCH  TO   * 

*  INTERPRETER  * 

*  FCR  FIRST   * 

*  INSTR.     » 

*  EXECUTION   * 


IS  THE 
SUPERVISOR 
CALL  CODE 
►    ZERO? 


*  MULTIPLY 

*  PROGRAM 

*  CONTROL 

*  COUNTER  BY 

*  FOUR 


•  • 
•OBTAIN  NUMBER* 
•OF  PROBLEM  TO» 

•  BE  GRADEO   • 

•  • 
•*••*•*••**•••* 


............... 


♦STORE  PROBLEM* 
*    NUMBER     » 


**•**•**••***** 


»»»»«»*«*«»*»* 


•  SET  GRADER 
•"IN  PROCESS" 


*••••********* 


PRINT- 
STUDENT  HAS 

NOT 
INITIAL! ZEO 


IS  THE  GRADER 
I  N? 


IS  GRADING  IN 
PROCESS? 


•*•*•*••*•••••• 

•  * 
•SAVE  ADDRESS  • 

- — >•   OF  ENTRY    • 

•  POINT  IN    * 

•  R9SAVE     • 
**••*•*•*•••••* 


IS  THE 

ADDRESS 
DEFINED? 


••••••••••••••a****** 

•       PRINT-        * 
-->    INITIALIZER 

•   ROUTINE  NOT   • 

AVAILABLE 

••**••*•**••••* 


*••*•••***••*•* 
♦ESTABLISH  REG* 
♦9  AS  BASE  FOR* 

♦  ANSWER     ♦ 

♦  SECTION    * 

♦  GINITOS  * 
•*••♦**•**••*•• 


»«*»»••»«»••»•»••«•«• 


PRINT  INITIAL 

-IZATION 

SUCCESSFUL 


**»«»*«»**»»**« 


•••*•*•**♦♦♦♦♦♦ 

EXIT  TO 
INTERPRETER 

••*••••*•***••* 


••**»**•••*•*•* 

•  CBTAIN     • 

*  ACDPESS  CF   * 

*  ANSWER     • 
•SECTION  FROM  • 

♦  R9SAVE     ♦ 
♦♦♦♦♦♦»»»***♦♦♦ 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

EXIT  TO  MAIN 

GPAOER 

*♦♦♦♦♦*****•*♦♦ 


•**♦♦♦♦•*••**•**♦♦♦♦♦ 

♦   PRINT  INITIAL   ♦ 

-IZATION 

♦      BEGUN      ♦ 

♦♦***••*♦♦♦♦♦♦♦ 


♦♦♦♦*••**••••** 

*  * 

*  GET  PROBLEM  • 
•NUMBER  AGAIN  * 
•C  MULT.  BY  4  • 

*  FOR  INDEX  • 
*•••*••»••••«»» 


•**•••*•••••••• 

•  LOAD  ENTRY   ♦ 

•  POINT  OF    • 

*  ANSWER     *- 
•SECTION  INTO  * 

*  REGISTER  9   • 
•*•*•♦♦♦♦♦••**• 


IS  THE  SVC 
CODE  VALID? 


••****••••***•* 

♦  RETURN  TO   ♦ 

♦  INTERPRETER  ♦ 

♦  FOR      ♦ 

♦  NON-GRAOING  ♦ 

♦  RUN      * 
**♦♦♦♦•••••»•»• 


**•♦*****♦•••*•***••* 

*      PRINT-       * 
— >    SUPERVISOR 

•  CALL  UNKNOWN   • 

••**•**•*•*•**♦ 


***•*•••*••••** 

EXIT  TO 

INTERPRETER 
***♦♦♦•♦•**••** 


AfiFNC 

V 

•»»*****«»*«»***»»»♦« 

•       PRINT-        * 

ABNORMAL  ENC 

•     OF  JOB      * 

*******••••*•** 


.«..«...».....< 


............... 


............... 

EXIT  TC  SOS 
CONTROL 
PRCf.RA" 

••»•«•*•»*•*♦•• 
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***  APPENDIX  C  —  CHART  NUMBER  3  *•• 
INTERMEDIATE  ANSWER  GRADING  ROUTINE 


«»***•«******«* 
START  INTANS1 

»**»***«**«**** 


ROUTINE 

SIMILAR  FOR 

INTERMEDIATE 

ANSWERS 

1,2,3.4.65 


*  OBTAIN 

*  STUDENT'S 

*  ANSWER 


*************** 

*  GET    ADDRESS    * 

*  OF    CORRECT      * 

*  ANSWER:         * 

*  (INSTRUCTOR  * 

*  DEFINED!    * 
*************** 


IS  THE 
STUDENT'S 

ANSWER 
CORRECT? 


DOES 

INSTRUCTOR 

HAVE  MESSAGE 

FOR  STUDENT?* 


*************** 

•  * 

•  GET  ADDRESS  * 
-->*     OF      * 

♦INSTRUCTOR'S  * 

•  MESSAGE    * 
«**•»*»*»*****« 


*  *NO 

*  CHECK  TO  SEE   * 
IF  MESSAGE  IS 

*  DEFINED?     * 


********************* 

*   PRINT--  TEXT    * 

OF  INSTRUCTOR 

*     MESSAGE     * 

*************** 


*    DOES     *NO 
INSTRUCTOR    * 
WANT  CORRECT 

DATA       * 
SUBSTITUTED?* 


********************* 

PRINT — 

*    INTERN.  ANS    * 

INCORRECT. 

*  DATA  SUB.  RUN  * 

CONT. 

•***••••****•** 


A******************** 

*      PRINT —      * 

INTERMEDIATE 

*     ANSWER      » 

CORRECT 

*************** 


*•*••*•*••*•*•* 

*  PUT  CORRECT  * 

*  ANSWER  IN   * 

*  STUDENT    *- 

*  ANSWER  AREA  * 

*  (REG  61    * 
**•***•*•*****• 


****••***•***** 

*  * 

*  BRANCH  TO   * 
•OBTAIN  CREDIT* 

*  FOR  CORRECT  * 

*  ANSWER     * 
*************** 


NODATA1  I 

V 

*****•**•*•*•*•*****« 

PRINT — 

*    INTERM.  ANS    * 

INCORRECT. 

*  DATA  NOT  SUB.  * 

CONT. 

*************** 


*************** 

EXIT  TC 

MACHGRD 
***•***••*****• 


CUT  I 
V 

*************** 

*  RETURN  TO   * 

*  INTERPRETER  * 

*  FOR  FURTHER  * 

*  PRCGRAM    * 

*  EXECUTION   * 
•••***•*•****•* 


*************** 

EXIT  TO 
INTERPRETER 

***•*••*••***** 
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•  ••  APMNOIX  C  --  CHART  NUMBER  4  ••• 
FINAL  ANSWER  GRADING  ROUTINE 


START  FINANS 


****»•**«****« 
♦GETANS 


*  OBTAIN 

*  STUDENT'S 

*  ANSWER 

••*•**•*•**••• 


*  GET  ADORESS  * 

*  OF  CORRECT   * 

*  ANSWER:  * 
»  (INSTRUCTOR  * 
»  DEFINEO)  * 
*************** 


IS  THE 

STUDENT'S 

FINAL  ANSWER 

*  CORRECT?   < 


ERR7 


PRINT —  FINAL 

ANSWER 

INCCRRECT 


•*•**»*»*••*••* 


********************* 

»   PRINT—  FINAL   * 

ANSWER 

*     CORRECT     * 

*************** 


***••••**•***** 

*  * 
♦CRFDIT  FIFTY  * 

*  POINTS  FOR   *- 
♦CORRECT  FINAL* 

*  ANSWER     * 
*••••*•*••**•*• 


**••**••*•**•*• 

*  * 

*  BRANCH  TC   * 
♦TOTAL  MACHINE* 

♦  GRACE  FOR   ♦ 

♦  PROGRAM    ♦ 
*••*•***•**•••• 


•***•••••*•••*• 

EXIT  TC 
GRADEIT 

•*•*»»••*•**••• 
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•**  APPENDIX  C  ~  CHART  NUMBER  5  •** 

ROUTINE  FOR  ACCUMULATING  INTERMEDIATE  ANSWER  CREDIT 


*************** 

START  — 
MACHGRD 

»»*»«*»»*•«»»♦* 


*  • 

*  LOAD      * 

*  ACCUMULATED  * 
*IIF  ANY!  SUM  * 
•INTO  REGISTER* 
*****•*»**»••** 


*  * 

*  ADD  TEN    * 

*  POINTS  TO   * 

*  ACCUMULATED  • 

*  SUM      * 


*************** 

*  * 

*  STORE     * 

*  ACCUMULATED  * 

*  CREDIT  BACK  * 

*  IN  SUM     * 


*************** 

*  * 

*  CONVERT  THE  * 

*  SUM  TC     * 

*  OECIMAL    * 

*  * 
*************** 


****** ********4 

*  4 

*  UNPACK  AND   < 

*  REMOVE  THE   < 

*  SIGN      « 

*  4 

****** «**»****< 


♦*•**»«»•»»**** 

*  MOVE  CREDIT  • 

*  FOR     * 
-- ^INTERMEDIATE  * 

•  ANSWER  INTO  * 

•  MESSAGE    * 


•     PRINT —     * 

MESSAGE  TO 

*  .    STUDENT     * 

««««*»*•«***»*• 


"ACCUMULATED 

INTERMEDIATE 

ANSWER 

CREDIT-   ■ 


**••*•****•***•****** 


PRINT—  LINE 
FOR  SPACING 


*•*•*******•*•* 


•**•***••**•**• 


•OBTAIN  RETURN* 
•    ADDRESS    * 


*************** 


••****•••**•*** 

*  * 

*  RETURN  TO   * 

*  INTERPRETER  * 

*  FOR  FURTHER  * 

*  EXECUTION   * 
•a************* 


*************** 

EXIT  TO 
INTERPRETER 

******•*••***•• 
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•  ••  APPENOIX  C  —  CHART  NUMBER  *  ••* 
ROUTINE  TO  OBTAIN  STUOENT'S  ANSWER 


ENTER  FROM 
GRACING 
SECTION 


••••»•♦•••♦•••• 

*  * 

*  GET  PROGRAM  * 

*  CONTROL    * 
•COUNTER  FRCN  » 

*  REGISTER  5   * 
♦••♦«••»•♦•»»•• 


............... 

»  » 

*  MULTIPLY  BY  * 

*  FOUR  TO    * 

*  OBTAIN  • 
•CORRECT  MCRC  • 
*************** 


•  LOAD  (SOS)   * 

*  AFTER  SVC  -  * 

♦  EFFECTIVE   * 

♦  AOORESS  CF   * 

*  ANSWER     * 
*************** 


*************** 

*  * 

*  MULTIPLY  eY  * 

*  FOUR  TO  GET  * 
»  360  AOORESS  * 

*  * 
*************** 


*************** 
•SAVE  AOORESS  * 
*IN  REGISTER  6* 

*  FOR  * 
♦SUBSTITUTION  * 

*  OF  OATA  * 
*************** 


*************** 

*  * 
♦OBTAIN  ANSWERS 
♦AND  STORE  IN  ♦ 
♦AREA  "STUANS"^ 

♦  ♦ 
*************** 


*************** 

*  • 

*  RETURN  TC  ♦ 
♦INTANSl.2,3,  ♦ 
♦4,5  OR  FINGRD^ 

*  * 
*************** 


***••*******•** 

EXIT  TO 

CALLER 

a************** 
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***  APPENDIX  C  —  CHAKT  NUMBER  7  *** 

TRANSLATION  AND  PRINTOUT  OF  MACHINE  GRADE  ROUTINE 

TRANS1,2,3 


**•*»**•«*****» 

START- 
TRANSLATION 

ROUTINE 
***••••»*•**••* 


*  • 
•CONVERT  GRADE* 

*  TO  PACKED   * 

*  DECIMALS  IN  * 

*  HORKHORD    * 

*************** 


*»»•*»*»**»««»« 


♦*»»*»»»*»»***» 


♦«**»*».»»**«* 


»  REMOVE  THE 
*     SIGN 


.............. 


*  • 

*  BLANK  GRACE  * 
♦FIELD  OF  THE  * 
»    MESSAGE    * 

*  * 
*************** 


*************** 

*  * 
*MCVE  GRADE  TC* 

*  IN-FIELD  CF  * 

*  THE  MESSAGE  * 

*  * 
*************** 


*************** 

*  * 

*  GIVE  THE    * 

*  PRINTER  THE  * 

*  ADORESS  OF   * 

*  THE  MESSAGE  * 
**••****•*****• 


********************* 

*     PRINT  THE     * 

MACHINE  GRAOE 

*     MESSAGE     * 

**•*•*•**••**** 


*************** 

EXIT  TO 
APPROPRIATE 

ROUTINE 
*•*******•**•** 
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•••    APPFND1X    C    —    CHART    NU«8FR    0    •*• 
RETURN    PPUTINFS 


START    RFTURN 

Tr  rrNTPOi 
pp.nr.R4H 


START    RETU°N 


♦LOAD    AOPRFSS  * 

•  FRO"    IMTHl  • 

•  ENTRY    TP  * 

•  GRADER  * 


............... 

*  • 

♦  LOAn    THE  * 

•  MVFO    APPPF?** 

•  TK'TO    REGI  STFR* 

*  14  * 


••»**•••«»*»»»••**•»• 


PRINT    ccace 
L  !NE 


»*••»**•»»«»*•• 


..................... 


or  iht    <:r/«cc 

L  [ME 


»**»»»*»♦***♦•* 


►  7FOT    PUT  • 

►  CPAPCR     IN  • 

►  rRrTF*«    /MP  * 

►  r.Q^FR     IN  * 

►  RJTS  » 


♦LOAD    AOOP    "E    * 
•INITIM      CMTRV» 

*  FPOM  * 

*  intqopct        , 

*  (SAVE214I       * 

*«•♦*•******•«. 


CO 
i     > 

c.o 


*  * 

*  PTTIJRN    TO     "lc» 

*  CONTROL  • 

*  FRPGO/SM    VI  I     * 

*  PCPKKFEPFP        * 
.....a......... 


•  • 

*  LOAP    THC  * 

*s*vep  »orRcsc* 

♦  INTO    B  =GI^T"« 

*  14  * 


............... 

«•  XI  T    *n 

PPPKKFFTR 

•  «<■«••♦»•»••»•• 


EXTT      TfJ 
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***  APPENOIX  C  --  CHART  NUMBER  9  *** 

TOTAL  MACHINE  GRAOE  ROUTINE  (FIRST  SECTION) 


♦»*»»*•♦**»»♦»» 

START  TOTAL 

MACHINE  GRAOE 

ROUTINE 


*  * 

*  LOAO  THE    * 

*  NUMBER  OF   * 
•INTERMEDIATE  * 

*  ANSWERS    * 

*************** 


*  *YES 

IS  THE  NO.  OF  * 
INTERMEDIATE     * 
ANSWERS  FIVE?  * 


****•*•***•*••* 

*  • 

*  OBTAIN  THE   * 
■ — >»  ACCUMULATED  » 

*  INT,  CREDIT  * 

*  (SUM)     * 


»«»»»»*«»«««»»« 


*************** 


*  *YES 

IS  THE  NUMBER  * 

OF  * 

INTERMEDIATE   * 
ANSWERS  >  5?* 


INTERMEDIATE 
ANSWERS  ZERO? 


*************** 

*  * 

*  MULTIPLY    BY    * 

*  FOUR    FOR  » 

*  INDEX  « 

*  * 
*************** 


*  * 

*  BRANCH    TO       * 
*GRD1    2    3    CR    <.*- 

*  ROUTINES         ♦ 

*  * 
*************** 


PRINT- 
*   INCORRECT  NO.   * 
-->        OF 

»  INTERMEDIATE   * 

ANSWERS 

*************** 


*************** 

EXIT  TO 

INTERPRETER 

•**•********•*• 


*******•*••***• 

*  STORE     * 
•INTERMEDIATE  * 

*  GRACE  IN    *- 

*  ACCUMULATOR  * 

*  (SUM)     * 
**•**••**•*•*** 


»***•***•****•* 

►  * 

►  BRANCH  TO   * 

►  TOTAL  THE   * 

►  GRADE     * 

►  * 
»*•**•*******•• 


*************** 

EXIT  TO 

FINGRD 

*•***•*•*••**•• 
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•••  APPENDIX  C  --  CHART  NUMtER  10  ••• 

TOTAL  MACHINE  GRAOE  ROUTINE  (SECOND  SECTIONI 


START-  SUM  OP 
TOTAL  GRAOE 


»•••«•••••••••• 


•ANSWER  CREDl 
•     ISUHI 


*  * 
•OBTAIN  FINAL  * 
•ANSWER  CREOIT* 

*  (SUM1)    • 

*  * 
*••••*•*•**•*•* 


**•*****••*•••• 


*  ADD  THE  TWO  • 

•  VALUES     • 


••••••*•»••••*• 


*•*••*•••**•••* 
•TRANS1         * 


•   TRANSLATE   • 
•GRADE  C  PRINT* 


•••••••*•••••** 


DCES 

INSTRUCTOR 

HAVE  A  FINAL 

•  MESSAGE?   ' 


GET  AOORESS 
OF  MESSAGE 


DID  THE 
INSTRUCTOR 
DEFINE  THE 
•  MESSAGE? 


••••ft**************** 

•      PRINT —      • 

MESSAGE  TO 

•   THE  STUDENT   • 

*******•**•••** 


■TOTAL 

MACHINE 

GRADE*    ' 


••*••*•*•****•* 

•  • 

•  BRANCH  TO   • 

•  HALT      * 

•  INSTRUCTION  • 

•  ROUTINE    • 
••••••••••••••a 


•*•••*♦•••••••• 

EXIT  TO 

FINOUT 

••••••••••••••a 


7H 


*•*  APPENOIX  C  —  CHART  NUMBER. 11  ••• 
INSTRUCTIONS  EXECUTED  ROUTINE 


***********•»•* 

START  INSTR. 

EXECUTED 

ROUTINE 

**♦••*•••*•**•• 


*************** 

*  * 

*  BLANK  MALT   * 

*  INSTRUCTION  • 
•MESSAGE  FIELD* 

*  * 
*************** 


*************** 

*  * 
♦MOVE  IN  FIRST* 
•PART  OF  HALT  * 

*  INSTRUCTION  • 

*  MESSAGE  * 
********••*•*** 


*************** 

*  • 
♦LOAD  VALUE  OF* 

*  CURRENT    * 

♦  INSTRUCTION  ♦ 

♦  COUNT     * 
♦♦***•**♦♦♦♦♦♦♦ 


****♦♦♦♦♦♦♦♦♦♦♦ 


UNPACK  THIS 
VALUE 


*************** 


*************** 

♦  * 
♦CBTA1M  LENGTHS 

♦  AND  PUT  THE  ♦ 

♦  NUMBER  IN   ♦ 

♦  MESSAGE    ♦ 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


♦♦♦♦♦♦♦♦*•***** 

♦  * 

♦  MOVE  IN  THE  ♦ 

♦  SECCND  PART  ♦- 

♦  OF  THE     * 

♦  MESSAGE    * 
*********•*♦♦♦♦ 


*************** 


•GIVE  PRINTER  • 
->*  ADDRESS  OF   * 

•  MESSAGE    * 

•  * 
*»******•*»**** 


****•**•••••****•**•• 

•      PRINT—      • 

MESSAGE  TO 

*    STUDENT     * 

••••*********** 


INSTRUCTIONS 
EXECUTEO' 


*************** 

*  * 

*  RETURN  FROM  ♦ 

*  PRINTER  AND  *- 
•CHECK  NO.  OF  * 
•INSTR.  EXEC.  * 
**♦♦♦♦♦♦♦♦***** 


•  STUDENT   *YES 

REQUIRE  MORE   • 
INSTRUCTION     < 
EXEC  THAN    * 

*  SPEC?    • 


***••*•*******••••••• 

•      PRINT —      • 

MESSAGE  TO 

*   THE  STUDENT   * 

*•*•*•*•****•*• 


'YOUR  PROGRAM 

REQUIRED 

FEWER 

INSTRUCTIONS 

THAN..' 


••*•*♦♦♦♦♦♦♦♦♦♦ 

♦  ♦ 

•  ADD  TEN  • 
•POINTS  TO  THE* 
♦TOTAL  MACHINE^ 
♦GRADE  — BONUS^ 
♦♦♦♦♦♦********* 


*************** 


**♦♦♦♦♦***•*****♦♦*♦• 


•TOTAL 

MACHINE  GRADE 

WITH  BONUS= 


*************** 


*****♦*•******•**••*• 

*      PRINT —      * 

MESSAGE  TO 

*   THE  STUDENT   * 

*•***•*•******• 


'YOUR  PROGRAM 

REQUIRED  MORE 

INSTRUCTIONS 

THAN...' 


*************** 

•  * 

*  RETURN  FROM  ♦ 

*  PRINTER  AND  * 
♦HEAD  FCR  HOME^ 

♦  • 
******•♦♦♦♦♦♦♦♦ 


FINOUT1 

V 
♦♦♦♦♦♦*♦******* 

*  * 
♦RETURN  TO  SOS^ 

*  CONTROL    * 

*  PROGRAM    * 

*  ♦ 
*************** 


*♦**•*♦♦♦♦**♦** 

EXIT  TO 

BOOKKEEPER 
**•*•******•*•* 
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•  *•  APPENDIX  C  —  CHART  NUMtER  12  *•• 
STRUCTURE  OF  GENERAL  COMMUNICATIONS  SECTION 


START  — 
GINITQS 

DSECT 


THE  STRUCTURE 

IS  DEFINED  IN 

ORDER  AS 


*  • 

*  DEFINE     * 

*  STORAGE  FOR  • 

*  ANSWERS    • 

*  • 


ANSI  OS  F 


............. 


*••*•*••**•** 


ANS3  DS  F 


* 

»   ANS4  DS  F 

* 


►   ANS5  OS  F   < 


■  •  •••••••  ...... 

>   ANSF  DS  F   < 


•  DEFINE 
>•  STORAGE  FOR 

•  THE  NO,  OF 
•INT,  ANSWER 


..: 


•  OEFINE  THE   * 
•VALUE  FOR  NO.* 

•  OF  INSTR    * 

•  EXECUTED    • 

•  CHECK     • 


••••••*••*•••• 

* 

*   NOX  OS  F 

* 

•*••*••*•••*•* 


•  • 

•  DEFINE     • 

•  INSTRUCTOR   * 
•CONTROL  BLCCK* 

•  STORAGE    • 
•*•*•***••**••* 


•••••******•••* 

*  * 

•  * 
•CONTROL  OS  H  •- 

*  • 

•  • 
••••*•**•*•**** 


•  OEFINE     • 
>•  STORAGE  FOR  * 

•THE  MESSAGES  • 

•  TO  STUDENT   • 


SIX  MESSAGES: 

1MESS1  THRU 

IMESS5  t 

IMESSF 


•••***••*•**••* 


•MACRO:  IMESS-* 
•  GMESS  L132-  • 


............... 


DEFINES: 

IMESS   DC 

ALK13I)  - 

LENGTH  OF 

LINE  -1 


DC  XLl'Ol*  - 

PRINTER 
CONTROL  CODE 


DC  CL132* 
-  MESSAGE 
AREA 


***••*••*•••••* 

•  THIS  • 
•STRUCTURE  IS  • 
•OVERLAVED  BY  * 

•  INSTRUCTOR   • 

•  CSECT  * 
••••*••***••••• 


END  OF  OSECT 
•*••**•**•••••* 
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APPENDIX  D 


Examples  of  Graded  Programs 


An  example  of  a  program  with  the  grader  off  is  first  shown 
Examples  of  graded  programs  are  then  given. 
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